Thermal infrared (IR) cameras capture images using light with wavelengths in the range of seven and a half to fourteen micrometers. A typical IR camera uses an infrared sensor (or detector) to detect infrared energy that is guided to the sensor through the camera's lens. IR cameras can be utilized for a variety of imaging applications including, but not limited to, object detection and segmentation, passive motion detection, night vision, thermal mapping, health care, building inspection, surveillance, and the like. Recently, an attempt has been made in the IR industry to integrate IR cameras in advanced driver assistance systems and autonomous vehicle systems.
The application, and hence the type of camera, may depend on the infrared spectrum used. The infrared spectrum lies outside of the visible light range and consists of a near infrared section (NIR), with wavelengths of 0.75-1.0 micrometers (μm); a short infrared section (SWIR) with wavelengths of 1-3.0 micrometers (μm); a mid-Infrared section (MIR), with wavelengths of 3.0-5.0 μm; and a far-infrared (FIR) section, with wavelengths of 7.5-14.0 μm.
One type of FIR sensor are uncooled bolometric detectors. They have a small form factor and can be typically mass-produced using low-cost technology. In a typical arrangement, an uncooled sensor does not require a cryocooler for proper operation, but does require a shutter for frequent calibration. A shutter is a mechanical element placed between the lens and the FIR sensor designed to block incident scene radiation typically used for image nonuniformity correction. Generally, a shutter is composed of a flat-bladed flag, a sleeve, and an arm that connects the sleeve to the flag. The flag opens and closes at predefined time intervals.
Each detector within the FIR sensor is very sensitive to temperature changes that may result either from thermal radiation of the outside scene, i.e., the image of interest, or from the internal, or ambient, camera radiation, causing parasitic ambient drift. Since each detector has a slightly different responsivity to ambient temperature, the ambient drift pattern adds a random offset to each pixel that persists between different video frames and changes as a function of ambient temperature. Ambient drift removal is one of the crucial steps in proper image processing of thermal cameras. Without properly and completely rectifying this ambient drift, the output image includes a residual unwanted noise pattern, or residual nonuniformity.
Even after ambient drift is removed from an image, there often remains residual noise present in the scene. This residual nonuniformity can often be visually discernible to a human eye. Removing or mitigating such noise can be difficult and complicated.
A shutter is often used during a flat-field correction (FFC) procedure to address ambient drift, as well as the residual nonuniformity. During the FFC, the shutter presents a uniform temperature source to the FIR sensor. By capturing the flat-field source radiation, the camera can update the offset correction coefficients, which results in a more uniform image. The duration of the FFC procedure is a few hundred milliseconds, during which the image captured just prior to the shutter blocking the field of view is frozen until the FFC process is completed, when the shutter is reopened. The shutter-based FFC must occur every few minutes for proper calibration.
While using a shutter may improve the quality and accuracy of the thermal image captured by an FIR sensor, having a blackout period of hundreds of milliseconds is not acceptable in certain applications. For example, using a shutter-based FIR camera in advanced driver assistance systems and autonomous vehicle systems can pose a high risk, as the camera must frequently freeze for a few hundred milliseconds, an unacceptable amount of down time for driving situations. In addition, shutters include moving parts that wear out over time. This may cause a camera to malfunction during use and shorten the life of the camera.
The FIR camera designed for advanced driver assistance systems and autonomous vehicle systems should meet additional constraints other than safety. Such constraints include a small form factor, accurate and low latency image processing, and low-power consumption. Moreover, the presence of a shutter requires a larger camera body and additional energy to power the shutter movement. Thus, currently available FIR cameras, and in particular, shutter-based FIR cameras, are not well adapted for automotive applications.
While there are some manufacturers who produce shutterless cameras where calibration data is used for scene-based nonuniformity correction, merely relying on calibration performed during the manufacturing process often leads to inaccurate results. This is because the output of a FIR senor, and hence the camera, also depends on the current internal (ambient) camera radiation, which is known as parasitic ambient drift. Additionally, applying corrections based on calibration tables fall short of sufficiently correcting for residual nonuniformity, which is difficult to model.
It would therefore be advantageous to provide a solution to correct residual nonuniformity present in images, after applying calibration tables and performing ambient drift correction, in shutterless FIR cameras.